Papers with KL regularization
Efficient KL Divergence Estimation via Truncated Top-K Integration for Large Language Models (2026.acl-long)
Copied to clipboard
| Challenge: | Existing methods for estimating KL divergence using only top-k tokens suffer from high variance or systematic bias. |
| Approach: | They propose a top-k Importance-weighted KL Estimator that exploits the Zipfian structure of language model distributions by integrating only the top-K tokens. |
| Outcome: | The proposed estimator outperforms existing estimators on multiple benchmarks while exhibiting lower variance. |
More Insightful Feedback for Tutoring: Enhancing Generation Mechanisms and Automatic Evaluation (2024.emnlp-main)
Copied to clipboard
| Challenge: | Incorrect student answers can be valuable learning opportunities provided that the student understands where they went wrong and why. |
| Approach: | They propose to use a KL regularization term to achieve more targeted input representations and a preference optimization step to encourage student answer-adaptive feedback generation. |
| Outcome: | The proposed model outperforms existing models in 3.3 METEOR points. |
A Few Bad Apples Spoil the Bunch: Preventing Global Entropy Collapse Driven by a Small Set of Tokens in LLM Reasoning (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning with verifiable rewards and Reinforced Learning from internal feedback fail to benefit from test-time compute due to entropy collapse and the resulting loss of reasoning diversity. |
| Approach: | They propose a strategy that assigns each generated token a redistribution score and applies selective KL regularization to only the top 5% of tokens under this score. |
| Outcome: | The proposed model improves on both RLVR and RLIF models on math reasoning benchmarks, showing that targeted entropy control at a vanishingly small subset of tokens is sufficient to sustain reasoning diversity and effective test-time scaling. |
Countering Reward Over-Optimization in LLM with Demonstration-Guided Reinforcement Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches address ROO by adding KL regularization, requiring computationally expensive hyperparameter tuning. |
| Approach: | They propose a reinforcement learning approach that leverages human demonstrations and a reward model to recalibrate the reward objective. |
| Outcome: | The proposed approach achieves comparable performance to carefully tuned baselines while mitigating ROO in three RL language tasks. |
MDP-GRPO: Stabilized Group Relative Policy Optimization for Multi-Constraint Instruction Following (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can follow many natural-language instructions, yet they remain brittle when a request bundles multiple explicit constraints, such as asking the LLM to respond in a particular structure with an exact ending phrase. |
| Approach: | They propose a method which stabilizes learning through multi-temperature sampling to increase reward dispersion, dual-anchor advantages to restore gradients in homogeneous groups, prospect-theoretic shaping to bound updates and penalize violations based on Kahneman Tversky’s theory and asymmetric KL regularization. |
| Outcome: | The proposed method outperforms standard GRPO on FollowBench, IFEval, and a curated multi-constraint dataset, improving strict constraint satisfaction by up to 5.0% on Llama-3.2-3B. |